Coadjoint orbits of the group UT(7, K)

Orbits of the coadjoint action play an important role in harmonic analysis, theory of dynamic systems, noncommutative geometry. A.A. Kirillov’s orbit method allows to reduce the classification problem of unitary representations of nilpotent Lie groups to the classification of coadjoint orbits [1, 2]. This makes possible to solve the problems of representation theory in geometric terms of the orbit space. But it turns out that the classification of coadjoint orbits is a difficult problem itself. In particular the classification problem of coadjoint orbits of the triangular group for an arbitrary dimension is far from its solution [3, 4]. The classification of regular orbits was achieved in the pioneering paper on the orbit method [2]. It is pointed out in the quoted papers that the case n 6 6 was considered in [5]. In the paper we consider the case of dimension 6 7 and subregular orbits of an arbitrary dimension. We check the hypothesis of the second author that the orbits may be described in terms of minors of the characteristic matrix. The paper consists of three sections. The first section contains the classification of coadjoint orbits of the triangular group for n 6 7 over an arbitrary field K of zero characteristic. The description is given in terms of admissible diagrams (Theorem 1.6). In the next Theorem 1.7 we solve the classification problem of coadjoint orbits for n 6 7 in terms of canonical forms. For any canonical form the polarization is constructed. This allows to classify unitary irreducible representations of the real triangular groups and absolutely maximal primitive ideals of the respective universal enveloping algebras (Theorem 1.9 and Corollaries). Note that this diagram method doesn’t work for the triangular group of an arbitrary dimension. Authors have constructed counterexamples to Theorems 1.6 and 1.7 for n = 9. Orbits of the coadjoint action of a nilpotent group are closed subsets ([6], 11.2.4). Suppose the ideal I(Ω) of K[g] = S(g) consists of functions that equal zero on the orbit Ω. This ideal is an absolutely maximal ideal (AMP-ideal) with respect to the natural Poisson bracket in S(g). The map Ω 7→ I(Ω) establishes the bijection between the set of coadjoint orbits and the set of AMP-ideals. In §2 we present the set of generators for an arbitrary AMP-ideal for n 6 7 (Theorem 2.3). This allows to represent an orbit as a set of solutions of polynomial equations. It turns out that one can present the system of generators as a system of some polynomials of the form P − c, where P is a coefficient of a minor of the characteristic matrix Φ(τ), c ∈ K. Authors believe that the last proposition (see Corollary 2.4) is true for all coadjoint orbits of the triangular group of an arbitrary dimension. The description of regular orbits is given in terms of minors of the matrix Φ ( [2] and Theorem 3.1). In §3 of the paper (Theorem 3.3) we present the classification of subregular orbits (i.e., orbits of